A First Empirical Evaluation Framework for Security Risk Assessment Methods in the ATM Domain

Evaluation and validation methodologies are integral parts of Air Traffic Management (ATM). They are well understood for safety, environmental and other business cases for which operational validation guidelines exist which are well defined and widely used. In contrast, there are no accepted methods to evaluate and compare the effectiveness of risk assessment practices for security. The EMFASE project aims to address this gap by providing an innovative framework to compare and evaluate in a qualitative and quantitative manner risk assessment methods for security in ATM. This paper presents the initial version of the framework and the results of the experiments we conducted to compare and assess security risk assessment methods in ATM. The results indicate that participants better perceive graphical methods for security risk assessment. In addition, the use of domain-specific catalogues of threats and security controls seems to have a significant effect on the perceived usefulness of the methods.